Processing of estimating a motion of each pixel between consecutive frames in a moving image is used in an MPEG (Moving Picture Experts Group) coding method, and device, a three-dimensional noise removal method, and device that remove noise by position-aligning images of a plurality of frames and combining the images, and a super resolution technique for generating a high resolution image from images of a plurality of frames.
Two luminance images f and f′ having a predetermined time interval therebetween, in a moving image including a minute motion, and a result obtained by estimating a motion vector from f to f′ in each pixel are illustrated in FIG. 1. Hereafter, a horizontal direction component of a motion vector at a coordinate (x, y) is denoted by u(x, y), and a vertical direction component thereof is denoted by v(x, y).
An example of a conventional motion vector estimation method is described in NPL 1. In the technique, an energy function E represented by the following Math. is considered.
Math. 1
  E  =            ∑              x        ,                  y          ∈                                          ⁢          f                      ⁢          {                                    (                                                                                f                    x                                    ⁡                                      (                                          x                      ,                      y                                        )                                                  ·                                  u                  ⁡                                      (                                          x                      ,                      y                                        )                                                              +                                                                    f                    y                                    ⁡                                      (                                          x                      ,                      y                                        )                                                  ·                                  v                  ⁡                                      (                                          x                      ,                      y                                        )                                                              +                                                f                  t                                ⁡                                  (                                      x                    ,                    y                                    )                                                      )                    2                +                  α          ⁡                      (                                                                                                  ▽                    ⁢                                                                                  ⁢                                          u                      ⁡                                              (                                                  x                          ,                          y                                                )                                                                                                              2                            +                                                                                      ▽                    ⁢                                                                                  ⁢                                          v                      ⁡                                              (                                                  x                          ,                          y                                                )                                                                                                              2                                      )                              }      
Here, the first term is called data term, and the second term is called regularizing term. Furthermore, fx(x, y), fy(x, y), and ft(x, y) are partial differentials of a pixel value f(x, y) at a coordinate (x, y) in directions of an x axis, a y axis, and a time axis, respectively, and are represented by the following Math.fy(x,y)=f(x,y+1)−f(x,y)fx(x,y)=f(x+1,y)−f(x,y)ft(x,y)=f′(x,y)−f(x,y)  Math. 2
Furthermore,∇u(x,y),∇v(x,y)  Math. 3are gradient vectors at the coordinate (x, y) and represented by the following Math.
Math. 4
            ▽      ⁢                          ⁢              u        ⁡                  (                      x            ,            y                    )                      =          (                                                                  u                ⁡                                  (                                                            x                      +                      1                                        ,                    y                                    )                                            -                              u                ⁡                                  (                                      x                    ,                    y                                    )                                                                                                                        u                ⁡                                  (                                      x                    ,                                          y                      +                      1                                                        )                                            -                              u                ⁡                                  (                                      x                    ,                    y                                    )                                                                        )                  ▽      ⁢                          ⁢              v        ⁡                  (                      x            ,            y                    )                      =          (                                                                  v                ⁡                                  (                                                            x                      +                      1                                        ,                    y                                    )                                            -                              v                ⁡                                  (                                      x                    ,                    y                                    )                                                                                                                        v                ⁡                                  (                                      x                    ,                                          y                      +                      1                                                        )                                            -                              v                ⁡                                  (                                      x                    ,                    y                                    )                                                                        )      
The first term in { } in the right side of Math. 4 is called data term and represents a constraint that the luminance value on the images f and f′ does not change between before and after a movement with motion vectors u(x, y) and v(x, y). In the same way, the second term is called smoothing term, and represent a constraint that the motion vectors u(x, y) and v(x, y) change with spatial smoothness. Intensities of both constraints are adjusted by using a smoothing term weight a.
Optimum motion vectors u(x, y) and v(x, y) minimize the above-described energy function. At this time, the following constraint Math. concerning u(x, y) and v(x, y) is obtained by making a partial differential of the above-described energy function with respect to u(x, y) and v(x, y) equal to zero.fs(x,y)2·u(x,y)+fx(x,y)·fy(x,y)·v(x,y)+fx(x,y)·ft(x,y)−α·Δu(x,y)=0fs(x,y)·fy(x,y)·u(x,y)+fy(x,y)2·v(x,y)+fy(x,y)·ft(x,y)−α·Δv(x,y)=0  Math. 5
Here, Δ is the Laplacian, and Δu(x, y) and Δv(x, y) are represented by the following Math.Δu(x,y)=u(x+1,y)+u(x−1,y)+u(x,y+1)+u(x,y−1)−4·u(x,y)Δv(x,y)=v(x+1,y)+v(x−1,y)+v(x,y+1)+v(x,y−1)−4·v(x,y)  Math. 6
The above-described constraint Math. includes equations concerning the motion vectors u(x, y) and v(x, y) at each coordinate, and u(x, y) and v(x, y) are found by solving the simultaneous equations.
By the way, since repetitive calculations need a long time, a technique for detecting a motion vector without using repetitive calculations is described in PTL 1.
A technique concerning a motion detection circuit is described in PTL 2. However, the technique only detects whether there is a motion and cannot detect a motion vector.
A technique of changing over a gradient method or a block matching method depending upon the number of detected gradient parts and detecting a motion vector is described in PTL 3. However, this technique finds only one motion vector for an image as a whole.
A technique of detecting a motion vector from a contracted image and then detecting a motion vector of an image having an original resolution is described in PTL4. However, this technique relates to improvement of a search range of a motion vector.
A technique of determining a hierarchy in which motion detection is started by using data obtained by conducting discrete wavelet decomposition on an image, in a method for hierarchically detecting a motion vector is described in PTL 5.